Image processing apparatus and image processing method

ABSTRACT

According to various embodiments, an image processing apparatus includes an inputting unit configured to receive input of an image, a range information acquiring unit configured to acquire range information about a subject for each region of the input image, and a converting unit configured to assign a gradation to each region of the input image based on the range information and convert luminance data of the input image according to the assigned gradation.

BACKGROUND

Technical Field

The present invention relates to an image processing technique forproviding an effect such as a shadow image to digital image data.

Description of the Related Art

There is an image processing method by which the amount of informationof input image data is decreased to emphasize a silhouette of subject.For example, an information processing circuit is proposed to generate acutout-style image by performing filling based on outlines extractedfrom input image data (see, for example, Japanese Patent ApplicationLaid-Open No. 2011-180643).

However, a case where two or more subjects overlap each other is notconsidered in the processing discussed in Japanese Patent ApplicationLaid-Open. No. 2011-180643. Specifically, in a case where outlinesextracted from overlapping subjects cross each other, there is apossibility that the subjects cannot be discriminated from each other.This possibility is conspicuous especially in a case where a shadowstyle image is generated simply by omitting outlines and performing thefilling to express the silhouette of a subject.

SUMMARY

According to various embodiments, an image processing apparatus includesan inputting unit configured to receive input of a captured image, anacquiring unit configured to acquire, for each image area of thecaptured image, range information indicating a distance from an imagingapparatus to a subject at the time of imaging, and a converting unitconfigured to convert, for each image area of the captured image, aluminance value of the captured image according to the distance to thesubject, and convert a color value of the captured image intopredetermined value, based on the range information, wherein theconverting unit converts the luminance value of the captured image suchthat the luminance value decreases as the distance to the subjectbecomes shorter.

Further features will become apparent from the following description ofexemplary embodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the configuration of an image processing apparatusaccording to a first exemplary embodiment.

FIG. 2 illustrates the configuration of a silhouette tone processingunit according to the first exemplary embodiment.

FIGS. 3A and 3B each illustrate input/output characteristics of a lookuptable (LUT) for use in gradation assignment according to the firstexemplary embodiment.

FIGS. 4A to 4F illustrate images having undergone respective steps inthe silhouette tone processing according to the first exemplaryembodiment.

FIG. 5 schematically illustrates the distance relationship between eachsubject according to the first exemplary embodiment.

FIGS. 6A to 6C are flow charts illustrating operations in silhouettetone processing according to the first exemplary embodiment.

FIG. 7 illustrates the configuration of a silhouette tone processingunit according to a second exemplary embodiment.

FIGS. 8A and 8B are flow charts illustrating operations in LUT selectingprocessing according to the second exemplary embodiment.

FIG. 9 illustrates an image having undergone silhouette tone processingaccording to the second exemplary embodiment.

FIG. 10 illustrates input/output characteristics of a LUT for use ingradation assignment according to the second exemplary embodiment.

FIGS. 11A and 11B schematically illustrate the distance relationshipbetween each subject according to the second exemplary embodiment.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

The following describes a first exemplary embodiment of the presentinvention.

In the present exemplary embodiment, an example will be described inwhich an exemplary embodiment of the present invention is applied to animage processing apparatus including as imaging system such as a digitalcamera and a scanner. This is not a limiting example, however, and anexemplary embodiment of the present invention is not limited to theexample but is applicable to any image processing apparatus capable ofprocessing image data. Specifically, the image processing apparatus maybe an information processing apparatus such as a personal computer, amobile information terminal, or an image forming apparatus such as aprinter. The foregoing also applies to other exemplary embodiments.

FIG. 1 is a block diagram illustrating the configuration of a digitalcamera that is an image processing apparatus 100 according to a firstexemplary embodiment.

In the image processing apparatus 100, light from the subject is focusedonto an image sensor 2 by an optical system 1 such as a diaphragm and alens, and photoelectrically converted into an electric signal, which isoutput from the image sensor 2. The image sensor 2 is, for example, asingle-panel color image sensor including a commonly-used primary colorfilter. The primary color filter includes three types of color filtershaving main transmission wavelength bands in the neighborhood of 650 nm,550 nm, and 450 nm, respectively, and configured to capture images ofcolor planes corresponding to red (R), green (G), and blue (B) bands.

In the single-panel color image sensor, the color filters are spatiallyarranged in the form of a mosaic for each pixel, and each pixel obtainsintensity on a single color plane, so that a color mosaic image isoutput from the image sensor 2.

An analog/digital (A/D) conversion unit 3 converts the electric signalacquired from the image sensor 2 into a digital image signal and outputsthe digital image signal to a development processing unit 4. In thepresent exemplary embodiment, 12-bit image data is generated for eachpixel at this point.

The development processing unit 4 performs a series of developmentprocessing such as pixel interpolation processing, luminance signalprocessing, and color signal processing on the digital image signaloutput from the A/D conversion unit 3. Through the processing performedby the development processing unit 4, the digital image signal isconverted from the RGB color space into the color space of 8-bitluminance (Y) data and chrominance (U, V) data, and output as YUV datafrom the development processing unit 4.

A range information acquisition unit 12 acquires range information abouteach pixel of the image data output from the development processing unit4. The range information in the present exemplary embodiment may be arelative distance from the in-focus position of an image to the subjector an absolute distance from an imaging apparatus to the subject at thetime of imaging. The absolute distance or relative distance may beeither one of the distance on the image plane side or the distance onthe object side. Further, the distance may be represented by thedistance in a real space or by the defocusing amount. Further, the rangeinformation is not limited to the example in which the distance is setfor each pixel of an image, and the distance may be set for each regionhaving a predetermined range of an image.

In the present exemplary embodiment, the range information acquisitionunit 12 acquires range information about the subject from the image dataoutput from the development processing unit 4. To acquire the rangeinformation, a publicly-known technique may be used such as a methoddiscussed in Japanese Patent Application Laid-Open No. 2000-156823 whichuses image plane phase-difference pixels, or a method using a pluralpieces of differently blurred image data which are captured undervarious imaging conditions (Depth From Defocus method (DFD method)).

The range information acquisition unit 12 may acquire the rangeinformation without using image data output from the developmentprocessing unit 4. For example, the range information acquisition unit12 may acquire the range information using a phase-difference detectionelement.

In the present exemplary embodiment, when an imaging mode in whichsilhouette tone processing is performed on a captured image is set withrespect to the image processing apparatus 100, a silhouette toneprocessing unit 5 performs silhouette tone processing, which will bedescribed below, on image data output from the development processingunit 4.

In the present exemplary embodiment, the image processing apparatus 100includes the imaging system, and the configuration of the optical system1 and the image sensor 2 functions as an image input unit to receiveinput of an image. In a case where an exemplary embodiment of thepresent invention is applied to an image processing apparatus includingno imaging system, an input interface configured to receive input of animage from the outside of the image processing apparatus 100 functionsas an image input unit.

A signal processing unit 6 performs resizing processing, etc. on imagedata having undergone the silhouette tone processing and supplies theimage data to an output unit 7. The output unit 7 performs at least oneof following steps: outputting to an output interface such as ahigh-definition multimedia interface (HDMI) (registered trademark),recording on a recording medium such as semiconductor memory card, andoutputting to a display apparatus (not illustrated) of the imageprocessing apparatus 100.

In a case where a normal imaging mode in which the silhouette toneprocessing is not performed on a captured image is set with respect tothe image processing apparatus 100, image data output from developmentprocessing unit 4 is input directly to the signal processing unit 6 asspecified by a broken line in FIG. 1.

A user interface (UI) unit 9 includes one or more input devices such asa switch, a button, and a touch panel provided in the display apparatus(not illustrated). An operation from the outside such as a userinstruction is input to the image processing apparatus 100 via the UIunit 9. A control unit 10 performs calculation in response to anoperation input via the UI unit 9 and controls each unit of the imageprocessing apparatus 100.

The control unit 10 controls one units via a bus 8 and performsnecessary calculation processing as needed.

A memory 11 stores image data for use in the processing units andinformation data at the time of capturing an image such as an aperturevalue, shutter speed, International Organization for Standardization(ISO) sensitivity, white balance gain value, and color gamut settingssuch as standard RGB (s-RGB). The stored data is read and used whenneeded in response to an instruction from the control unit 10. Further,as illustrated in FIG. 1, the components of the image processingapparatus 100 are connected together such that the components cancommunicate with each other via the bus 8.

The following describes an image processing method, according to oneembodiment, in the silhouette tone processing to be executed by theimage processing apparatus 100 and the configuration of an imageprocessing circuit that realizes the image processing method, withreference to FIG. 2. The silhouette tone processing unit 5 is configuredto apply features of a shadow image as image effects to image data. Asused herein, representative features of a shadow image refer to asilhouette expression with the inside of an outline filled with a singlesolid color, a blurring amount corresponding to the distance from thescreen, a peripheral portion where the light is significantly reduced,and number limit of colors.

In the present exemplary embodiment, gradations of luminance (Y) dataare determined using the range information corresponding to the capturedimage, so that even in a case where a main subject overlaps with anothersubject which is located in front of or behind the main subject existingat an in-focus position, effects of a shadow image expressingsilhouettes of the respective subjects can be achieved.

A gradation assigning unit 201 assigns gradations to YUV image datainput from the development processing unit 4. As to a gradationassigning method in the present exemplary embodiment, gradationsdetermined based on the range information input from the rangeinformation acquisition unit 12 and a one-dimensional lookup table (LUT)208 are assigned to luminance (Y) data, and a predetermined value (e.g.,0) is uniformly assigned to chrominance (UV) data.

The LUT 208 is a LUT selected by the LUT selecting unit 206 based on thesilhouette tone type, from LUTs 207 a and 207 b provided for eachsilhouette tone type and having one of the characteristics illustratedin FIGS. 3A and 3B. The silhouette tone types will be described below.

A blur image generating unit 202 generates a blur image by performingblurring processing (smoothing processing) such as filter processingusing a low-pass fitter on image data to which the gradations of thesilhouette tone are assigned. As used herein, the blur image refers toan image that is blurred compared to the input image, i.e., an imageobtained by eliminating higher frequency components than a predeterminedfrequency from the input image.

There are several possible methods for performing the blurringprocessing. For example, there is a method in which a low-pass filterusing Gaussian filter coefficients is applied vertically andhorizontally to an image to smooth the image by a single operation.However, in order to realize a blur level expected in the silhouettetone processing through one smoothing processing, a large kernel size isrequired of low-pass filter, which leads to a significantly large amountof processing time. In other words, it is not so realistic to performthe processing on hardware of the camera. Thus, in the present exemplaryembodiment, a reduction process circuit and an enlargement processcircuit are used in combination to generate a blur image in order toshorten the processing time and acquire a desired blur. Details ofoperations in the blur image generating processing will be describedbelow with reference to a flow chart illustrated in FIG. 6C.

A combining unit 203 combines an image input from the gradationassigning unit 201 with a blur image input from the blur imagegenerating unit 202 under a specific condition. A shadow image can beobserved by placing an object between a screen and a light source toproduce a shadow and then projecting the shadow of the object onto thescreen with the light source. However, the shadow image has such acharacteristic that the definition of outlines varies according to thedistance between the object and the screen.

To provide the foregoing characteristic to the image data, the imagedata to which the gradations of the silhouette tone are assigned by thegradation assigning unit 201 expresses a distance to the subject by thegradations, according to the present exemplary embodiment. Thus, thecombining unit 203 replaces with a blur image a region having a valueequal to or larger than a predetermined value in the image data inputfrom the gradation assigning unit 201, whereby the characteristic of ashadow image that the blur amount varies according to the distance fromthe screen can be achieved as an image effect.

Further, a marginal illumination reduction unit 204 performs processingon the image data to which the blur effect of the silhouette tone isgiven as though the marginal illumination of the image data isdecreased. In order to generate a distinct shadow, an object to form ashadow and a screen are illuminated by a point light source, so that ashadow image has a characteristic that a single point on the screen hasthe highest brightness and the brightness decreases as distance from thepoint becomes larger.

To provide the foregoing characteristic to image data, processing forreducing the marginal luminance of the image data is performed with thecenter of the screen (image) being the point having the highestbrightness in the present exemplary embodiment. Specifically, the imagedata is multiplied by marginal luminance reduction data (marginalillumination reduction data) of a two-dimensional distributioncorresponding to the image data to adjust the luminance distribution ofthe image. The processing for reducing the marginal luminance of imagedata is not limited to the foregoing processing. Specifically, to adjustthe luminance distribution, the luminance of the data may be reduced bydividing the image data by the luminance reduction data or adding orsubtracting the luminance reduction data to or from the image data maybe used.

Further, a method is applicable to an exemplary embodiment of thepresent invention, in which the luminance distribution of image data isadjusted by calculation instead of preparing the marginal luminancereduction data in advance. A light source subject such as the sun can beexpressed by placing the point having the highest brightness not in thecenter of the screen but in an upper or bottom end portion or outsidethe screen. In this case, marginal coordinates of the illuminationreduced data may be vertically and horizontally shifted, and thenmultiplied by the marginal illumination reduced data.

A toning unit 205 performs toning on image data having undergone themarginal illumination reduction processing. A basic shadow image is ablack/white monotone image which is generated by illuminating acolorless screen with light emitted by an incandescent lamp, a lightemitting diode (LED) bulb, a projector light source, etc. and recognizedas colorless light by human eyes. However, the toning can be performedby inserting a color film in front of the light source to express thesky blue or the sunset red.

To give the foregoing characteristic to the image data, YUV image datainput to the toning unit 205 is converted into the RGB space by a matrixcalculation, and each RGB plane is multiplied by different coefficients,in the present exemplary embodiment. For example, in order to adjust thecolor to a color the sky blue, the processing specified by the formulasbelow may be performed. Naturally, the coefficient may be set to one ina case where no toning is performed.

R′=R×218/255

G′=G×255/255

B′=B×191/255

The toning unit 205 re-converts the color-adjusted RGB image data intothe YUV format by a matrix calculation and outputs the data as YUV imagedata to the signal processing unit 6.

The setting of the luminance value (Y) by the gradation assigning unit201 and the setting of the color value (UV) by the toning unit 205 areseparately performed to enable conversion into a monotone image showingthe distance of the region in a single-color gray scale, for each regionof the image.

The following describes in more detail the gradation assigningprocessing performed by the gradation assigning unit 201 in the presentexemplary embodiment.

The gradation assigning unit 201 assigns the gradations of the imagedata according to the range information input from the range informationacquisition unit 12, but the range information cannot be used direct asthe gradations of the image data because various forms of rangeinformation are conceivable as described above. Thus, LUTs respectivelycorresponding to the forms of range information to be used are stored inadvance on the memory 11, and a result of applying the LUT to the rangeinformation is assigned as the gradations of the image data.

In the present exemplary embodiment, the range information is used thatexpresses the distance to the subject from each pixel image data in 256gradations, where 0, 128, and 255 are an infinite end, a focal plane,and a close end, respectively.

FIG. 3A is the LUT 207 a for converting the range information into thegradations of the silhouette tone. A shadow image has distinctivegradation characteristics that all major subjects are shadows, abackground is slightly bright in contrast to the shadows, and a regionwhere no subject exists is the screen to project the shadow image andthus has the highest brightness. A background portion in a shadow imageis brighter than a shadow because an object to form the shadow islocated away from the screen and close to a light source to allow lightfrom the light source to come around the object.

To give gradations close to a shadow image to the image data, the LUT207 a uniformly gives a gradation value of 0 to an input value of 128 orgreater, i.e., a subject existing between the main subject at the focalplane and the closest distance end, whereby the subject is expressed asa shadow. Next, an input value of 0, i.e., a subject located between theinfinite end and an input value of 128, is given a linearly-interpolatedgradation value between 220 and 0 based on the input value, whereby thesubject is expressed as a background and a screen.

FIGS. 4A to 4F illustrate images (data) having undergone the steps inthe silhouette tone processing performed by the silhouette toneprocessing unit 5 in the present exemplary embodiment.

FIG. 4A illustrates a sample of image data containing YUV data outputfrom the development processing unit 4 and input to the silhouette toneprocessing unit 5. FIG. 5 schematically illustrates the distancerelationship between the subjects in the image data. In the image data,a person that is the main subject exists in the center of the screen atthe focal plane, and a tree trunk standing on the left hand side of thescreen exists at the closest distance end, as illustrated in FIGS. 4Aand 5. Further, buildings and a forest exist more distant than the focalplane, and the sky exists at the infinite end, as illustrated in FIGS.4A and 5.

FIG. 4B illustrates an image of the range information output from therange information acquisition unit 12 and input to the silhouette toneprocessing unit 5. In FIG. 4B, the value of the sky existing at theinfinite end is zero, and the value of the buildings and the forestexisting more distant than the focal plane is 64. The value of theperson existing at the focal plane is 128, and the value of the treetrunk existing at the closest distance end is 255. A ground existingbetween the person and the tree changes continuously between 128 and255.

FIG. 4C illustrates an image of the image data output from the gradationassigning unit 201. In FIG. 4C, as a result of the gradation assigningprocessing described above, the value of the person existing at thefocal plane and the values of the tree, the ground, etc. located closerto the closest distance end than to the focal plane and having a largevalue of range information are uniformly zero, and the person, the tree,the ground, etc. are represented as shadows to emphasize thesilhouettes. On the other hand, the values of the buildings and theforest located closer to the infinite end than to the focal plane andhaving a smaller value of range information are uniformly 200, wherebythe buildings and the forest are discriminable from the shadow regionswhile the silhouettes of the buildings and the forest are emphasized.Further, the value of the sky existing at the infinite end uniformly220, and the sky has the highest brightness in the screen and isexpressed as a screen in the shadow image.

FIG. 4D illustrates an image of the image data output from the combiningunit 203. In the present exemplary embodiment, a region of the imageoutput from the gradation assigning unit 201 that has a value of 200 orgreater, i.e., background region, is replaced with the blur image outputfrom the blur image generating unit 202. From FIG. 4D it can berecognized that while the sharpness of the outlines of the buildings andthe forest decreases, the high sharpness of the outlines of the shadowsis maintained, whereby the silhouettes of the shadows are moreemphasized.

FIG. 4E illustrates an image of the image data output from the marginalillumination reduction unit 204. In the present exemplary embodiment,the marginal illumination reduction processing is performed bymultiplying the image data output from the blur image generating unit202 by the marginal luminance reduction data for concentrically reducingthe input values in such a way that the center of the screen is reducedby 100% of the input value, each of the four corners of the screen isreduced by 30% of the input value, and other regions are reduced bypredetermined percentages of the input values. From FIG. 4E it can berecognized that the reduction in the marginal illumination is expressedas though the screen is illuminated by a point light source.

As described above, in a final image acquired as a result of executionof the gradation assigning processing using the characteristic specifiedin the LUT 207 a in FIG. 3A, the gradation value of the main subjectexisting at the focal plane is set to zero which surely expresses themain subject as a shadow, so the expression is similar to the actualshadow image. However, since the gradation value of a subject closer tothe imaging plane than to the main subject is also set to 0, in a casewhere, for example, the person in FIG. 4E moves to overlap with the treetrunk, the silhouettes of the person and the tree trunk is notdiscriminable. Thus, control may be selectively performed to separatethe gradations of the main subject and the subject closer to the imagingplane than to the main subject according to the user's drawing plan.Accordingly, in the present exemplary embodiment, an appropriate LUT canbe selected and applied according to the user's drawing plan, which willbe described later.

FIG. 6 is a flow chart illustrating entire operations in the silhouettetone processing performed by the silhouette tone processing unit 5illustrated in FIG. 2. Each operation in the flow chart is performed bythe control unit 10 or by in each unit according to an instruction fromthe control unit 10.

In step S601, the LUT selecting unit 206 selects and sets the LUT 208used by the gradation assigning unit 201.

In step S602, the gradation assigning unit 201 performs gradationassigning processing as described above according to the selected LUT208.

In step S603, the blur image generating unit 202 performs blur imagegenerating processing on the image data to which the gradations areassigned.

In step S604, the combining unit 203 performs combining processing asdescribed above on the blur image output from the blur image generatingunit 202 and the image data output from the gradation assigning unit201.

In step S605, the marginal illumination reduction unit 204 performsmarginal illumination reduction processing as described above on theimage data having undergone the combining processing.

Alternatively, only one of the blurring processing in step S604 and themarginal illumination reduction processing in step S605 may be appliedto the captured image having undergone the gradation conversion. Evenwhen only one of them is applied, more effect of the silhouette tone canstill be provided.

In step S606, the toning unit 205 performs toning processing asdescribed above on the image data having undergone the marginalillumination reduction processing, and the image data is output from thesilhouette tone processing unit 5. Then, the processing is ended.

The following describes in detail the LUT selecting processing in stepS601 in FIG. 6, with reference to the flow chart illustrated in FIG. 6B.As described above, in the present exemplary embodiment, an appropriateLUT is selected and applied according to the user's drawing plan. Forexample, in a case where the user's drawing plan is to reliablydetermine the silhouette of the main subject existing at the focalplane, the gradation assigning processing is performed using a LUT 2having the characteristic that the gradation value of the closestdistance end takes a gradation value of zero and the main subject takesa gradation value higher than zero, as illustrated in FIG. 3B. On theother hand, in a case where the user's drawing plan is to express themain subject as a silhouette as in a shadow image, the gradationassigning processing is performed using a LUT 1 having thecharacteristic that a gradation value of 0 is uniformly given to asubject existing between the main subject at the focal plane and theclosest distance end, as illustrated in FIG. 3A. The LUTs 1 and 2 bothhave the characteristic that a gradation value of 220 is given to asubject existing at the infinite end to represent the subject as ascreen. Further, a gradation value which is linearly interpolatedbetween the infinite end and the main subject based on the distance isgiven to a subject existing between the infinite end and the mainsubject to represent the subject as a background.

The gradation values set in the LUTs 1 and 2 illustrated in FIGS. 3A and3B are mere examples, and the gradation values are not limited to thegradation values specified in the LUTs 1 and 2.

FIG. 4E illustrates the output image from the silhouette tone processingunit 5 in the case where the LUT 1 is used as the LUT 208 used by thegradation assigning unit 201 as described above. On the other hand, FIG.4F illustrates an output image in the case where the LUT 2 is used asthe LUT 208. In FIG. 4F, while the tree trunk is expressed as a shadow,the person is expressed in halftones, so the silhouettes of the tree andthe person are clearly discriminable from each other.

In the present exemplary embodiment, prior to the generation of an imagein silhouette tone, the user can select, as a silhouette tone type,whether to prioritize a shadow image style or the determination ofsilhouette. A silhouette tone type input by the user via the UI unit 9is recorded on the memory 11.

In step S6011, the control unit 10 reads from the memory 11, forexample, the silhouette tone type that prioritizes a shadow image style.

Next, in step S6012, the LUT selecting unit 206 determines the readsilhouette tone type and selects the corresponding LUT 208 from the LUTs207 stored on the memory 11 for each silhouette tone types. In the caseof the silhouette tone type that prioritizes the shadow image style, theLUT 1 is selected. In the case of the silhouette tone type thatprioritizes the determination of silhouette, the LUT 2 is selected. TheLUTs 207 for the respective silhouette tone types are stored in advanceto avoid a huge amount of calculation processing at the time of theimaging, whereby high-speed continuous imaging can be performed withoutdecreasing the imaging frame rate.

In step S6013, the control unit 10 sets the selected LUT 208 to thegradation assigning unit 201, and the processing returns to the mainprocessing.

The following describes in detail the blur image generating processingin step S603 in FIG. 6, with reference to the flow chart in FIG. 6C. Asdescribed above, in the blur image generating processing, a blur imageis generated by combining the reduction processing and the enlargementprocessing. More specifically, the reduction processing is performed onimage data which has undergone the gradation assigning processing toreduce the amount of information, and then the enlargement processing isperformed together with interpolation to give the image a blur effect.

First, the reduction size of a most-reduced image is set according tothe target blur size. For example, in the present exemplary embodiment,the size of each side of a blur image which replaces an infinite endregion is one-fourth the size of each side of an input image (the numberof pixels is one-fourth in the vertical direction and in the horizontaldirection). In steps S6021 to 6024, in a case of reducing the size ofeach side by one-fourth, the reduction by one-half is vertically andhorizontally repeated N times (in this case N=2). At this time, in stepS6022, a low-pass filter (LPF) having filter coefficients of [1, 2, 1]is applied vertically and horizontally to perform smoothing prior to thereduction in order to prevent an occurrence of folding of high-frequencycomponents, i.e., an occurrence of so-called moiré, that is caused bythe reduction. When the reduction processing is repeated N times andcompleted, then in steps S6025 to S6027, the enlargement processing isperformed until the size is enlarged to the original size. As in thereduction processing, the enlargement processing is repeated doubly Ntimes vertically and horizontally.

While the zoom ratio in the reduction is set to one-half in oneprocessing according to the present exemplary embodiment, the zoom ratiomay be one-quarter or is not limited to these. However, the filtercoefficients of the low-pass filter to be applied need to be changed asappropriate to prevent an occurrence of moiré. For example, in the casewhere the zoom ratio is set to one-quarter, the filter coefficients needto be set to [1, 4, 6, 4, 1].

As the foregoing describes, in the present exemplary embodiment, animage in silhouette tone is generated by generating luminance datahaving gradations of silhouette tone using range informationcorresponding to the input image, and then combining the luminance datawith black/white monotone data or color data having undergone toning toacquire final image data, in the silhouette tone processing forproviding the effect of the silhouette tone. In this way, even if thesubjects overlap one above the other with each other, the silhouettetone processing for expressing the silhouettes of the respectivesubjects can be realized.

Further, in the present exemplary embodiment, the gradation assigningprocessing is performed as appropriate according to the silhouette tonetype, so that an image in silhouette tone reflecting the user's drawingplan can be generated.

The following describes a second exemplary embodiment. As describedabove, in the silhouette, the brightness of a formed shadow variesaccording to the distance between an object forming the shadow and thelight source. In a case where a plurality of objects is placed atdifferent distances from the light source to generate a shadow image, aplurality of shadows different in brightness is generated, and thebrightness of the shadows of each object has a characteristic that thebrightness is discrete according to the distance from the light source.

While the LUT having a characteristic that continuous gradations areassigned based on the range information is used in the gradationassigning processing performed by the gradation assigning unit 201 inthe first exemplary embodiment, the characteristics of LUTs stored aspreset LUTs are discretized to calculate a gradation assignment LUT atthe time of the gradation assigning processing performed by thegradation assigning unit 201 in the second exemplary embodiment.

FIG. 7 is a block diagram illustrating details of the silhouette toneprocessing unit 5 according to the second exemplary embodiment.Processing performed by a block having the same reference numeral as inFIG. 2 are similar to that in FIG. 2, therefore, its description isomitted. A difference from the first exemplary embodiment is that a LUTdiscretizing unit 701 is included to analyze imaging conditions receivedfrom the control unit 10 and calculate an appropriate gradationassignment LUT 702 corresponding to the imaging conditions.

FIG. 8 is a flow chart illustrating an example of operations in the LUTselecting processing in step S601 in FIG. 6A according to the presentexemplary embodiment. The rest of the silhouette tone processing issimilar to the operations illustrated in FIG. 6.

In step S801, the control unit 10 reads as the silhouette tone type oneof the modes of prioritizing a shadow image style and determining thesilhouette. In the description below, the mode in which the silhouettedetermination is prioritized is read from the memory 11, as an example.

In step S802, the LUT selecting unit 206 determines the read silhouettetone type and selects a corresponding LUT 208 from the LUTs 207 for eachsilhouette tone type stored in the memory 11. In the case where thesilhouette determination is prioritized, the LUT 2 is selected.

In step S803, the LUT discretizing unit 701 performs discretizingprocessing on the selected LUT to calculate the gradation assignment LUT702.

In step S804, the control unit 10 sets the discretized gradationassignment LUT 702 to the gradation assigning unit 201, and theprocessing returns to the main processing.

FIG. 9 illustrates an image having undergone the silhouette toneprocessing which is performed by the silhouette tone processing unit 5according to the present exemplary embodiment. In a comparison betweenFIGS. 4F and 9, while the gradation of the ground changes continuouslyfrom the base of the tree to the feet of the person in FIG. 4F, agradation value of 100 is uniformly assigned to the person and theground near the person and a gradation value of zero is uniformlyassigned to the tree at the closest distance end and the ground near thetree in FIG. 9. The gradation is discretized according to the distanceso that the characteristic of the shadow image described above isexpressed more appropriately.

The following describes in detail the LUT discretizing processing instep S803 in FIG. 8A, with reference to the flow chart in FIG. 8B.

According to the present exemplary embodiment, the LUT 207 convertsrange information into gradations of silhouette tone. The LUTdiscretizing unit 701 divides the LUT 207 into four regions that are amain subject existing near the focal plane, a subject at close range, abackground subject, and a subject at the infinite end. The samegradation value is given to a region having a predetermined distancerange, whereby the range information is discretized into fourgradations. FIG. 10 illustrates the gradation assignment LUT 702calculated by discretizing the LUT 207 b. The LUT 207 b is chosen in thecase of the silhouette tone type that prioritizes the silhouettedetermination.

A threshold value for determining that a subject is close to thein-focus position at a predetermined distance or much closer will bedenoted by TH_close, whereas a threshold value for determining that asubject is far from the in-focus position at a predetermined distance ormuch farther will be denoted by TH_far. The gradation assignment LUT 702has the following characteristics. A gradation value of 100 representinga main subject is given to 128 representing the focal plane, the inputwithin a range between 128−TH_far and 128+TH_close being a main subjectregion. A gradation value of zero representing a shadow is given, theinput greater than 128+TH_close being a closest subject. As to the inputsmaller than 128−TH_far, a gradation value of 220 representing a screenis given, the input in the neighborhood of zero being a subject at theinfinite end, and a gradation value of 200 representing a background isgiven, other input being a background subject. Specifically, a gradationvalue is assigned to each region depending on where the distance of theregion is included, in a plurality of distance ranges set stepwise basedon the distance from the focal plane.

In this case, the value of the range information that is input from thegradation assignment LUT 702 varies depending on the imaging conditionat the time of capturing an input image that is a target of thesilhouette tone processing. Thus, in the present exemplary embodiment,as described below, the threshold value for the LUT discretization isadaptively controlled according to the imaging condition. Specifically,the plurality of distance ranges described above is set based on theimaging condition of the input image.

In the present exemplary embodiment, the control unit 10 can acquirefrom the optical system 1 the shortest possible imaging distance.Further, the distance of the focal plane can be acquired from the rangeinformation acquisition unit 12.

In step S8011, the control unit 10 reads from the memory 11, forexample, the shortest possible imaging distance of 1 m and the distanceof the focal plane of 11 m as the imaging conditions. In the presentexemplary embodiment, an image processing apparatus including an imagingsystem is used, so the imaging conditions set at the time of capturingthe input image are stored on the memory 11, and the foregoingprocessing is realized by reading the stored imaging conditions.

In step S8012, the LUT discretizing unit 701 calculates a thresholdvalue necessary for the LUT discretization based on the imagingconditions. In the present exemplary embodiment, for example, the LUTdiscretization is performed using a subject existing in front of orbehind the focal plane within 1 m from the focal plane as a mainsubject.

schematically illustrates the distance relationship between the subjectsin the case where the shortest possible imaging distance of 1 m and thedistance of the focal plane of 11 m are set as the as imagingconditions. In FIG. 11A, TH_close 1101 and TH_far 1102 for the LUTdiscretization are both, for example, 15.

FIG. 11B schematically illustrates the distance relationship between thesubjects in a case where the shortest possible imaging distance of 1 mand the distance of the focal plane of 6 m are set as the imagingconditions. Since the distance of the focal plane is shortened to 6 m,the value of range information for representing the same distancebecomes relatively large in a region closer to the imaging plane than tothe focal plane. Thus, the threshold value for the discretization needsto be increased in front of or behind the focal plane within 1 m fromthe focal plane, so that TH_close 1103 under the imaging conditions inFIG. 11B is, for example, 30. On the other hand, the value of rangeinformation for representing the same distance becomes relatively smallin a region more distant from the imaging plane than the focal plane.Thus, the threshold value for the discretization needs to be decreasedin front of or behind the focal plane within 1 m from the focal plane,so that TH_far 1104 under the imaging conditions in FIG. 11B becomes,for example, eight.

While the shortest possible imaging distance of the optical system 1 andthe distance of the focal plane are described as examples of the imagingconditions in the present exemplary embodiment, the imaging conditionsare not limited to the described examples. For example, the focal lengthor the aperture value may be used as the imaging conditions. When thefocal length or the aperture value is changed, the maximum distancewithin which range information can be acquired, i.e., the distance ofthe infinite end, changes, so that the value of range information forrepresenting the same distance relatively changes in a region moredistant from the imaging plane than from the focal plane. For example,if the distance of the infinite end is increased, TH_far can bedecreased. On the other hand, if the distance of the infinite end isdecreased, TH_far can be increased.

Further, the threshold values may be set by combining elements cited asexamples of the imaging conditions to determine the discretized distancerange.

In step S8013, the LUT discretizing unit 701 performs LUT discretizingprocessing as described above using the threshold values calculated instep S8012 to discretize the LUT 207 and calculates the gradationassignment LUT 702, and the processing returns to the main processing.

As described above, in the present exemplary embodiment, an image insilhouette tone is generated by generating luminance data havinggradations of silhouette tone using range information corresponding tothe input image and then combining the luminance data with black/whitemonotone data or color data having undergone toning to acquire finalimage data in the silhouette tone processing for giving the effect ofthe silhouette tone. In this way, even if a subject overlaps withanother subject one above the other, the silhouette tone processing forexpressing the silhouettes of the respective subjects can be realized.

Further, in the present exemplary embodiment, the LUT is discretizedusing the threshold values appropriate for the imaging conditions, andthe gradation assigning processing is performed using the discretizedLUT, whereby the taste of the silhouette tone can be expressed to amaximum extent.

While the hardware configurations of the respective blocks of thesilhouette tone process unit 5 are described in the above exemplaryembodiments, each operation of the respective blocks can be realized bysoftware, so that a part or all of the operations of the silhouette toneprocessing unit 5 may be implemented as software processing. Similarly,a part or all of the other blocks of the image processing apparatus 100in FIG. 1 may be implemented as software processing.

The gradation assigning unit 201 performs gradation assigning processingusing the one-dimensional LUT has been described as an example in eachof the exemplary embodiments described above. The method for thegradation assigning processing is not limited to the methods describedabove, and any method may be used as long as the gradation assigningprocessing is performed with the characteristics illustrated in FIG. 3A,3B, or 10, such as calculating an output pixel value by calculation.

Other Exemplary Embodiment

An exemplary embodiment of the present invention may be realized bysupplying a program for realizing one or more functions of theabove-described exemplary embodiments to a system or apparatus via anetwork or storage medium and causing one or more processors of acomputer of the system or apparatus to read and execute the program.Further, an exemplary embodiment of the present invention may berealized by a circuit (e.g., application specific integrated circuit(ASIC)) that realizes one or more functions.

While exemplary embodiments have been described, it is to be understoodthat the invention is not limited to the disclosed exemplaryembodiments. The scope of the following claims is to be accorded thebroadest interpretation so as to encompass all such modifications andequivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2015-234313, filed Nov. 30, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: aninputting unit configured to receive input of a captured image; anacquiring unit configured to acquire, for each image area of thecaptured image, range information indicating a distance from an imagingapparatus to a subject at the time of imaging; and a converting unitconfigured to convert, for each image area of the captured image, aluminance value of the captured image according to the distance to thesubject, and convert a color value of the captured image into apredetermined value, based on the range information, wherein theconverting unit converts the luminance value of the captured image suchthat the luminance value decreases as the distance to the subjectbecomes shorter.
 2. An image processing apparatus comprising: aninputting unit configured to receive input of a captured image; anacquiring unit configured to acquire, for each image area of thecaptured image, range information indicating a distance from an imagingapparatus to a subject at the time of imaging; and a converting unitconfigured to convert, for each image area of the captured image, thecaptured image into a shadow image of the subject expressed in light anddark according to the distance to the subject, based on the rangeinformation, wherein the converting unit converts the captured imagesuch that darkness increases as the distance to the subject becomesshorter.
 3. The image processing apparatus according to claim 1, furthercomprising a detecting unit configured to detect a subject from thecaptured image, wherein the converting unit assigns an equal luminancevalue to an image area included in a predetermined distance rangeincluding the subject detected by the detecting unit.
 4. The imageprocessing apparatus according to claim 1, wherein the converting unitassigns the luminance value to an image area depending on where adistance of the image area is included, in a plurality of distanceranges set stepwise based on a distance from a focal plane at the timeof the imaging.
 5. The image processing apparatus according to claim 1,wherein the converting unit converts the luminance value such that aluminance value of zero is uniformly assigned to a subject locatedcloser than a focal plane at the time of the imaging.
 6. The imageprocessing apparatus according to claim 1, further comprising a blurringprocessing unit configured to perform blurring processing on thecaptured image converted by the converting unit in a blur amount thatdiffers according to the distance to the subject, for each image area.7. The image processing apparatus according to claim 1, furthercomprising a marginal illumination reduction processing unit configuredto perform marginal illumination reduction processing on the capturedimage converted by the converting unit to reduce an amount of light in amarginal area of the captured image.
 8. The mage processing apparatusaccording claim 4, wherein the plurality of distance ranges is setaccording to an imaging condition at the time of the imaging.
 9. Theimage processing apparatus according to claim 8, wherein the imagingcondition includes at least one of a distance to the focal plane, adistance at which imaging is performable, a focal length, and anaperture value.
 10. An image processing method comprising: receivinginput of a captured image; acquiring, for each image area of thecaptured image, range information indicating a distance from an imagingapparatus to a subject at the time of imaging; and converting, for eachimage area of the captured image, a luminance value of the capturedimage according to the distance to the subject, and converting a colorvalue of the captured image into a predetermined value, based on therange information, wherein the converting converts the luminance valueof the captured image such that the luminance value decreases as thedistance to the subject becomes shorter.